Virtual three-dimensional objects in a live video

ABSTRACT

A method for depicting a three-dimensional object in a real environment, depicted in a final video played back on a monitor, includes the steps of generating at least one object video by filming the object with a camera from different filming directions for generating an object video, wherein the angle of an optical axis of the camera in relation to a central axis of the object is in each case determined for the respective filming direction, so that an angle can be associated with each filming direction. The method also includes generating a live video with at least one live camera and simultaneously inserting the object video into the live video at a defined position in the environment for generating the final video, and changing the position of the live camera in the environment and simultaneously depicting the object in the final video in a desired perspective in the environment.

TECHNICAL FIELD

The disclosure relates to a method for depicting a three-dimensionalobject in a real environment, depicted in a final video played back on amonitor.

BACKGROUND

In television and internet video productions, virtual objects areincreasingly embedded into a video that is broadcast live. The viewer isunable at first glance to distinguish between the real live video andthe inserted virtual object. Virtual insertions are almost no longerrecognizable as such particularly if the camera remains stationaryduring live production, or while the live video is being generated.However, problems arise when the position of the camera changes duringrecording, because the perspective of the inserted object must alsochange accordingly.

For example, images or videos of people are often prepared and placed byinsertion in the room or studio as a two-dimensional image or video. Forthe viewer, this then results in an image depicted on his monitor ortelevision set, which is largely formed from the live video into which,however, the two-dimensional person is also inserted.

However, if the camera recording the live image is moved so that it nolonger looks frontally at the inserted person, the images always appearnarrower. In the end, the inserted person would no longer be seen if thecamera moves around them by 90° and is disposed to the side of them.

To avoid this effect, there is the option of optically co-rotating theimage of the inserted computer graphics when the camera moves around it.This prevents the image from appearing narrower due to the new angle;however, it is then only seen straight from the front, i.e.two-dimensionally, from all perspectives.

It is also known to first capture objects, e.g. people, by means of athree-dimensional scan. In the process, the depth of the object is alsomeasured at every camera shot in order to finally be able to prepare athree-dimensional model of the object. In a next process step, thedifferent shots of the object from the respective perspectives areoverlaid (mapped) so as to result in the three-dimensional body. Thus,in principle, the individual shots are in this case “spanned”, piece bypiece, onto the body. However, this is done portion by portion, whereinthe transitions cannot always be displayed flawlessly. Thethree-dimensional shape of the object is also not perfect in this case,which can often be discerned in faces in such shots; so-called artifactsare produced. Notwithstanding the above, the generation of suchthree-dimensional scans is complex and requires a large data volume;manual post-processing is also necessary.

Another issue arises if the object is a person, for example, and issupposed to move in the future live video. In that case, thethree-dimensional scan also has to be animated, for which purposeknowledge or information, e.g. about joints and possible directions ofmovement, have to be taken into account. Also, such an animation israrely perfect; generally, it is clearly recognizable that this is ananimated video; the movement often appear unnatural.

SUMMARY

The present disclosure embeds a virtual object into a generated livevideo in as visually attractive a manner as possible. In particular,changes of perspective by moving the camera are supposed to be possiblewithout the inserted object video appearing unnatural or being distortedtoo much. However, it is also essential that the method can be carriedout as easily as possible. It is supposed to require as small a datavolume as possible and, in particular, necessitate no manualpost-processing, or only to a minor extent.

According to the disclosure, the advantage is achieved with a methodhaving the method steps of the independent claim.

Accordingly, the core idea of the disclosure is that, first, an objectvideo is generated which shows the object from different angles. Forthis purpose, the object is thus first shot with a filming camera fromdifferent angles or filming directions. The respective position of thefilming camera relative to the object is detected and stored in theprocess.

The term filming direction relates to a central axis of the object,which may be the vertical axis upwards or the horizontal transverseaxis, but also any other central axis passing through the object.

In a second method step, this object video is loaded into a computergraphic and depicted on a rectangle (i.e. only two-dimensionally asvideo texture), for example, which is positioned in the environment,hereinafter referred to as the studio. The studio is filmed with a livecamera during the recording, and a live video is generated. In thelatter, the object appears to the viewer virtually at a certain locationin space. Of course, the environment does not have to be an enclosedstudio; the live video may also be generated at any outdoor location.

The video that the observer can perceive thus includes the live videowith the embedded object video. For simplification purposes, this videothat the observer can perceive will be referred to as the final video.The object video is adapted to the position of the live camera by meansof a data processing device commonly used for this purpose.

What is important is that the relative angle between the object and thefilming camera is known while filming the object, and that the videosequence filmed at this relative angle can later be determined.

In a particularly simple variant, the object may be placed on a rotarytable for this purpose, which rotates during the filming process. Thefilming camera, which is then stationary, films the object, e.g. aperson, preferably continuously over 360°. Thus, the filming processthus takes place one-dimensionally; the object is only rotated about itsvertical or upward axis. The perspective or relative angle can beassociated with a certain video sequence by means of a correspondingmeasuring device on the rotary table, for example. An encoderdetermining the respective degree of rotation is a possible option forthis. A linear association may also be carried out by counting theindividual pictures that were shot and then correlating them with thecorresponding degree value.

In a particularly advantageous, simple variant, the rotated and filmeddegree value of the rotary table, most frequently 360 degrees, isdivided by the number of video images shot, which yields the angle pervideo image. Though the calculation result is slightly less exact thanthe result from an encoder, it simplifies the structure and issufficient for most desired applications. This method can also be usedfor other perspectives, e.g. in the case of a rotation of the objectabout its horizontal axis, or when filming the static or unmoving objectby moving the filming camera about the vertical axis of the object.

In a particularly simple variant, the rotary table may have a visiblescale on its outer face, which is also filmed during the filmingprocess. It is thus possible to see, in the shot object video, fromwhich perspective the object was filmed at a certain point in time, orwhich degree value had been reached at this point in time. In this case,the object video is embedded without the scale into the live video;however, the stored information apparent from the scale is neverthelessused as a video attribute for the desired depiction.

In an alternative variant, the object is positioned in a stationary andimmovable manner. In that case, the filming camera changes its position;for example, it circles once around the object over 360° and films itcontinuously in the process. Analogously to the above-mentioned variantwith the rotary table, the relative position or relative angle betweenthe filming camera and the central axis of the objective are permanentlydetermined and stored also in this case.

Both variants generate an object video showing the object from differentdirections, preferably across the full 360°.

The essential point is that the position of the recording live camera isalso determined permanently. Commonly used so-called tracking systemscan be used for this purpose. If the relative angle now changes betweenthe inserted object, i.e. the rectangle or the object video showntherein, then, according to the disclosure, that video sequence of theobject video is in each case played back that corresponds to therelative angle of the live camera to the virtual object. Thus, if thelive camera moves around the virtual object, the latter is readjusted byplaying back the object video in a targeted manner, and always appearsin the correct perspective. For this purpose, the object video is playedback forwards or backwards accordingly. In the final video that isshown, the object, e.g. a person, thus always appears as the observerwould see it in the studio if it were real.

The method according to the disclosure can be improved further by theobject video being shot with as high a resolution as possible. It isthus possible to show even partial areas of the object clearly duringthe later playback, by zooming in.

In contrast to the prior-art methods, the object may also move duringfilming, which is advantageous particularly when filming of people. Aturn of the head or an eye movement, for example, but in the end alsomore complex movements may be filmed. Though they are always the samewhen reproduced in the final video, this results in a very naturalmovement, however, in contrast to the known scanned and animated figuresor methods. For example, if the person rotates about their own verticalaxis while the filming camera moves around them, then attention shouldbe paid, when recording the live video, that this movement of the livecamera corresponds to the movement of the filming camera. Otherwise, themovement would be executed more slowly or also rapidly, and thus appearunnatural.

In principle, shooting several objects at the same time is alsopossible, in order to show natural overlapping or shadowing.Alternatively, different objects may also be shot individually and latercombined in the object video.

According to the disclosure, tilting the rotary table during filming isalso possible, in order thus to set up a different filming angle to thefilming camera.

As an alternative to arranging the object on a rotary table rotatingabout its vertical axis, according to the disclosure, it may also bearranged on a horizontal shaft which, accordingly, rotates about thehorizontal axis. This may be advantageous in the case of objects, forexample, that are supposed to also be observed from above. In thiscontext, pieces of furniture, building models, plants and the like areconceivable.

According to the disclosure, the objects can thus be attached to arotating horizontal shaft and filmed by a stationary camera. As analternative, it is of course also possible to guide the filming cameraaround a stationary object in a vertical plane.

The distance of the filming camera from the object may also be detectedand stored during filming, as an additional dimension. It is thuspossible, even while generating the live or final video, to change thedistance with the live camera from the virtual object, because theobject video can be adapted accordingly.

According to the disclosure, in a particularly simple variant of themethod according to the disclosure, the detection of the distance of thefilming camera can be dispensed with. The object video, or the videotexture (the region in the final video onto which the video is spanned,so to speak) is disposed in a stationary manner in the environment. Ifthe live camera moves away, this movement is measured by the cameratracking system, and the virtual scene, i.e. the object video or theregion onto which it is spanned, is shifted accordingly. It thenautomatically becomes smaller or larger because the live camera moves,just like a natural object. Since only a portion of the image isobtained, the perspective is slightly compressed or stretched; thus, theobject is thus provided with a slightly flatter or steeper impression ofdepth. Though the perspective of the object is in that case not thecorrect one, but since the depth of the object, for instance in the caseof humans, is very small, this is hardly perceived by the viewer; theeye is very undiscerning in this regard. It is thus possible to furtherreduce the complexity and amount of data.

This also applies to the height of the camera, which may also be changedand measured during the generation of the object video. Accordingly,this permits a change in the height of the live camera while generatingthe final video.

Though it is possible in the end, according to the disclosure, to filmthe object from all directions in order to also depict all directions ina future final video. However, it has proved to be particularlyadvantageous if a shot is created that is as simple as possible,preferably one-dimensional, because particularly the data volume is thuskept small. Most studio shots are generated in a one-dimensional manneranyway; the live camera usually remains at the same height during theshot.

In another advantageous variant of the disclosure, it is also possibleto also generate the object video live and insert it as a second livevideo into the first live video shot by the live camera. In this case,the position of the live camera is measured with a suitable cameratracking system during the live shot in the studio.

Instead of searching for the appropriate point in the video, the rotarytable is rotated, i.e. controlled, depending on the position of the livecamera. At the same time, i.e. live, the object or person is located onthe rotary table. A special advantage is that the person is able to movelive, and also react to the live events; in the process, the rotarytable always rotates the person into the appropriate direction, whichprescribes the position of the live camera. Thus, the object video isnot temporarily stored but also broadcast live as a second live videowithin the first live video.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is explained further with reference to the followingFigures. They are only supposed to illustrate the principle of thedisclosure, but are not supposed to limit the disclosure to theexemplary embodiments shown. In the Figures:

FIG. 1 : shows a rotary table with an object and a filming camera,

FIG. 2 : shows an apparatus for horizontally rotating the object and afilming camera,

FIG. 3 : shows an environment for generating a final video,

FIG. 4 : shows a simplified illustration of the final video from a firstperspective, and

FIG. 5 : shows a simplified illustration of the final video from asecond perspective.

DETAILED DESCRIPTION OF THE DRAWINGS

It is noted that the Figures are drawn in a very simplified manner; inparticular, they are not to scale.

FIG. 1 illustrates a first option for generating an object video. Anobject 20 to be filmed is disposed on a rotary table 22. A measuringdevice for detecting a rotation of the rotary table 22, i.e. the angleof rotation, is provided on the rotary table 22. In this case, themeasuring device 24 is connected to a data processing device that is notdepicted.

A filming camera 26 is disposed in a stationary manner and films theobject 20 while the rotary table 22 rotates and its rotation is detectedby the measuring device 24. In this way, an object video 28 is generatedwhich shows the object 20 from different filming directions. In theillustrated exemplary embodiment, the vertical axis Y-Y forms a centralaxis to which the filming direction relates. Basically, the rotary table22 may be turned by only a few degrees in the process, but also by 360°or even 720°; in the end, this depends on the requirements for theobject video 28 to be generated. The rotation by 720° has the effectthan no transition flaws or seams are created as one moves completelyaround the object.

FIG. 2 shows an alternative in which the object is fixed to ahorizontally orientated shaft 30. The filming camera 26 is alsostationary and disposed at a fixed height above the object 20. Ofcourse, the filming camera 26 may also be positioned underneath or tothe side of the object. In this exemplary embodiment, the vertical axisX-X forms the central axis to which the filming direction relates.

FIG. 3 shows an exemplary environment 32 in which a live video 34 is tobe generated. The environment usually is a studio for a TV production. Alive camera 36 generates the live video 34 and films a person 38 and anitem 40, depicted as a table. The live camera 36 can be moved within theenvironment 32, i.e. change its position.

FIGS. 4 and 5 illustrate a generated final video 42, which is formed ofthe recorded live video 34 and the object video 28. The object video 28is embedded into the live video 34, so that to the viewer, the object 20appears stationary within the environment 32. For example, such a finalvideo 42 is broadcast to television sets via a TV network.

FIGS. 4 and 5 illustrate the effect that is created when the live camera36 changes its position within the environment 32. The perspectivechanges with respect to the object 20, which nevertheless is depicted ina correct perspective.

The disclosure is not limited to the above-described and illustratedembodiments, but also includes other options that can be realized on thebasis of the disclosure. For example, several object videos 28 may alsobe embedded into the final video 42. For example, two or more objectvideos 28, which are also shot live, may be embedded, e.g. severalindividual people.

The invention claimed is:
 1. A method for depicting a three-dimensionalobject in a real environment, depicted in a final video played back on amonitor, the method including the following steps: generating at leastone object video by filming the object with a filming camera fromdifferent filming directions for generating an object video, wherein theangle of an optical axis of the filming camera in relation to a centralaxis of the object is in each case determined for the respective filmingdirection, so that an angle can be associated with each filmingdirection, generating a live video with at least one live camera andsimultaneously inserting the object video into the live video at adefined position in the environment for generating the final video, andchanging the position of the live camera in the environment andsimultaneously depicting the object in the final video in a desiredperspective in the environment, by the respective position of the livecamera in the environment being determined while shooting the livevideo, a video sequence of the object video is shown that alwayscorresponds to the relative angle between the live camera and theobject, such that the object video is adapted to the determined positionof the live camera, showing the object in the final video from thedirection that approximately corresponds to the direction of the livecamera towards the desired position of the object in the environment. 2.The method according to claim 1, wherein the filming of the objectincludes the following steps: arranging the filming camera in astationary manner, and rotating the object about its vertical axis. 3.The method according to claim 2, wherein the filming of the objectincludes the following steps: positioning the object on a rotary table,rotating the rotary table and simultaneously filming the object with thefilming camera, and determining the filming direction by determining thedegree value of the rotation of the rotary table.
 4. The methodaccording to claim 1, wherein the filming of the object includes thefollowing steps: arranging the filming camera in a stationary manner,and rotating the object about its horizontal axis.
 5. The methodaccording to claim 1, wherein the filming of the object includes thefollowing steps: arranging the filming camera in a stationary manner,and simultaneously rotating the object about its vertical axis and itsvertical axis.
 6. The method according to claim 1, wherein the filmingof the object includes the following steps: arranging the object in astationary and static manner, and filming the object whilesimultaneously moving the filming camera in a horizontal plane aroundthe object.
 7. The method according to claim 1, wherein the filming ofthe object includes the following steps: arranging the object in astationary and static manner, and filming the object whilesimultaneously moving the filming camera in a vertical plane around theobject.
 8. The method according to claim 1, wherein the filming camerais moved by at least 360° around the object while filming.
 9. The methodaccording to claim 1, wherein the position of the filming camera duringfilming is varied such that the object is filmed from all directions.10. The method according to claim 1, wherein the filming directions ofthe filming camera towards the central axis of the object are stored,and the object video is adapted to the determined position of the livecamera in the environment by playing back the video sequence of theobject video, which shows the object from the direction thatapproximately corresponds to the direction of the live camera towardsthe desired position of the object in the environment.
 11. The methodaccording to claim 1, wherein dividing the rotated and filmed degreevalue of the object by the shot number of video images as a basis forfinding the video sequence showing the object in the desired perspectivein the final video.
 12. The method according to claim 1, wherein the atleast one object video is embedded as a second live video into the firstlive video.